
<!DOCTYPE html>
<html lang="">


<head><meta name="generator" content="Hexo 3.8.0">
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, user-scalable=no">
  <meta name="theme-color" content="#202020">
  <meta http-equiv="x-ua-compatible" content="ie=edge">
  <script src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js" async></script>
  
  
    <meta name="keywords" content>
  

  
    <meta name="description" content="爬链家二手房数据">
  
  
  
  <link rel="icon" type="image/x-icon" href="/images/footer-logo.png">
  
  <title>爬链家二手房数据 [ 51AIOps 专注于运维自动化  微信： kaipython ]</title>
  
    <!-- stylesheets list from config.yml -->
    
      <link rel="stylesheet" href="//cdn.bootcss.com/pure/1.0.0/pure-min.css">
    
      <link rel="stylesheet" href="/css/xoxo.css">
    
  
</head>


<body>
  <div class="nav-container">
    <nav class="home-menu pure-menu pure-menu-horizontal">
  <a class="pure-menu-heading" href="/">
    
    <span class="title" style="text-transform:none">51AIOps 专注于运维自动化  微信： kaipython</span>
  </a>

  <ul class="pure-menu-list clearfix">
      
          
            
              <li class="pure-menu-item"><a href="/" class="pure-menu-link">首页</a></li>
            
          
      
  </ul>
   
</nav>

  </div>

  <div class="container" id="content-outer">
    <div class="inner" id="content-inner" style='margin-left:-68px!important'>
      <div class="post-container">
  <article class="post" id="post">
    <header class="post-header text-center">
      <h1 class="title">
        爬链家二手房数据
      </h1>
      <span>
        
        <time class="time" datetime="2017-04-07T12:55:09.000Z">
        2017-04-07
      </time>
        
      </span>
      <span class="slash">/</span>
      <span class="post-meta">
      <span class="post-tags">
        
      </span>
    </span>
      <span class="slash">/</span>
      <span class="read">
      <span id="busuanzi_value_page_pv"></span> 点击
    </span>
      <span class="slash">/</span>
    </header>

    <div class="post-content">
      <h4 id="前言"><a href="#前言" class="headerlink" title="前言"></a><div id="title">前言</div></h4><blockquote>
<p><font color="red" size="10px">侠之大者，为国接盘</font></p>
</blockquote>
<h4 id="准备工作"><a href="#准备工作" class="headerlink" title="准备工作"></a><div id="title">准备工作</div></h4><p><strong>首先你的环境中，应该会有如下的配置</strong></p>
<ul>
<li>urllib</li>
<li>BeautifulSoup</li>
<li>pandas</li>
</ul>
<p><strong>如果都没有的话，使用<code>pip3 install xxx</code>安装，不会的话google</strong><br><strong>最终引入如下的库</strong></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib.request</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br></pre></td></tr></table></figure>

<h4 id="接盘侠之路"><a href="#接盘侠之路" class="headerlink" title="接盘侠之路"></a><div id="title">接盘侠之路</div></h4><h5 id="1-设置请求固定页"><a href="#1-设置请求固定页" class="headerlink" title="1.设置请求固定页"></a>1.设置请求固定页</h5><p>设置列表页URL的固定部分，这次主要抓取链家二手房数据</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">url=<span class="string">'http://bj.lianjia.com/ershoufang/'</span></span><br></pre></td></tr></table></figure>

<h5 id="2-抓取数据"><a href="#2-抓取数据" class="headerlink" title="2. 抓取数据"></a>2. 抓取数据</h5><p>开始抓取数据，总共应该会有100页，因此我们使用一个循环，一页一页的进行请求，我们请求使用的是python内置的urllib库，因此最终的代码应该如下，应该不用我解释了吧</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1</span>,<span class="number">2</span>):</span><br><span class="line">    a = (url + <span class="string">'pg'</span> + str(i) + <span class="string">'/'</span>)</span><br><span class="line">    rsp = urllib.request.urlopen(url=a,data=<span class="literal">None</span>,timeout=<span class="number">10</span>)</span><br><span class="line">    page = rsp.read()</span><br><span class="line"></span><br><span class="line">    <span class="comment">#每次间隔0.5秒</span></span><br><span class="line">    time.sleep(<span class="number">0.5</span>)</span><br></pre></td></tr></table></figure>

<p><strong>注意：啪啪的时候一定要注意休息一下，再继续，所以time.sleep(0.5)你懂得</strong></p>
<h5 id="3-分析数据"><a href="#3-分析数据" class="headerlink" title="3. 分析数据"></a>3. 分析数据</h5><p>接下来，就已经有了数据了，我们开始分析数据，之前有介绍过bs4吧，因此此处我们使用这个库来进行爬取数据</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">lj = BeautifulSoup(page,<span class="string">'html.parser'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 分析出总价，放到一个列表中</span></span><br><span class="line">price = lj.find_all(<span class="string">'div'</span>,attrs=&#123;<span class="string">'class'</span>:<span class="string">'priceInfo'</span>&#125;)</span><br><span class="line">tp = []</span><br><span class="line"><span class="keyword">for</span> a <span class="keyword">in</span> price:</span><br><span class="line">    totalPrice = a.span.get_text()</span><br><span class="line">    tp.append(totalPrice)</span><br><span class="line">print(tp)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 分析出房屋具体信息</span></span><br><span class="line">houseInfo = lj.find_all(<span class="string">'div'</span>, attrs=&#123;<span class="string">'class'</span>: <span class="string">'houseInfo'</span>&#125;)</span><br><span class="line">hi = []</span><br><span class="line"><span class="keyword">for</span> b <span class="keyword">in</span> houseInfo:</span><br><span class="line">    house = b.get_text()</span><br><span class="line">    hi.append(house)</span><br><span class="line">print(hi)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 分析出房屋的关注量</span></span><br><span class="line">followInfo = lj.find_all(<span class="string">'div'</span>, attrs=&#123;<span class="string">'class'</span>: <span class="string">'followInfo'</span>&#125;)</span><br><span class="line">fi = []</span><br><span class="line"><span class="keyword">for</span> c <span class="keyword">in</span> followInfo:</span><br><span class="line">    follow = c.get_text()</span><br><span class="line">    fi.append(follow)</span><br><span class="line">print(fi)</span><br></pre></td></tr></table></figure>

<h5 id="4-清洗数据-整合"><a href="#4-清洗数据-整合" class="headerlink" title="4. 清洗数据+整合"></a>4. 清洗数据+整合</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 创建数据表</span></span><br><span class="line"><span class="comment"># house = pd.DataFrame(&#123; "houseinfo":hi, "followinfo":fi, "totalprice":tp&#125;)</span></span><br><span class="line">house = pd.DataFrame(&#123;<span class="string">"totalprice"</span>:tp&#125;)</span><br><span class="line"><span class="comment"># print(house)</span></span><br><span class="line"><span class="comment"># exit()</span></span><br><span class="line"><span class="comment"># 查看数据表内容</span></span><br><span class="line"><span class="comment"># house.head()</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 对房源信息进行分列</span></span><br><span class="line">houseinfo_split = pd.DataFrame((x.split(<span class="string">'|'</span>) <span class="keyword">for</span> x <span class="keyword">in</span> hi),columns=[<span class="string">'xiaoqu'</span>,<span class="string">'huxing'</span>,<span class="string">'mianji'</span>,<span class="string">'chaoxiang'</span>,<span class="string">'zhuangxiu'</span>,<span class="string">'dianti'</span>])</span><br><span class="line"><span class="comment"># print(houseinfo_split)</span></span><br><span class="line"><span class="comment"># houseinfo_split.head()</span></span><br><span class="line"></span><br><span class="line">house = pd.merge(house,houseinfo_split,right_index=<span class="literal">True</span>, left_index=<span class="literal">True</span>)</span><br><span class="line">print(house)</span><br><span class="line"><span class="comment"># house.head()</span></span><br><span class="line"></span><br><span class="line">huxing = house.groupby(<span class="string">'huxing'</span>)[<span class="string">'huxing'</span>].agg(len)</span><br><span class="line">print(huxing.head())</span><br></pre></td></tr></table></figure>

<p><strong>以上就是简单的数据分析，但是看了这么多，感觉怎样呢</strong><br><img src="https://ooo.0o0.ooo/2017/05/12/5915387e5b09d.jpg" alt="house.jpg"></p>

    </div>

  </article>
  <div class="toc-container">
    
  <div id="toc" class="toc-article">
    <strong class="toc-title">目录</strong>
    <ol class="toc"><li class="toc-item toc-level-4"><a class="toc-link" href="#前言"><span class="toc-text">前言</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#准备工作"><span class="toc-text">准备工作</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#接盘侠之路"><span class="toc-text">接盘侠之路</span></a><ol class="toc-child"><li class="toc-item toc-level-5"><a class="toc-link" href="#1-设置请求固定页"><span class="toc-text">1.设置请求固定页</span></a></li><li class="toc-item toc-level-5"><a class="toc-link" href="#2-抓取数据"><span class="toc-text">2. 抓取数据</span></a></li><li class="toc-item toc-level-5"><a class="toc-link" href="#3-分析数据"><span class="toc-text">3. 分析数据</span></a></li><li class="toc-item toc-level-5"><a class="toc-link" href="#4-清洗数据-整合"><span class="toc-text">4. 清洗数据+整合</span></a></li></ol></li></ol>
  </div>


  </div>
</div>
<script type="text/javascript" src="//rf.revolvermaps.com/0/0/8.js?i=5sr5du46f27&amp;m=0&amp;c=ff0000&amp;cr1=ffffff&amp;f=arial&amp;l=33" async="async"></script>
<div class="copyright">
    <span>本作品采用</span>
    <a href="https://creativecommons.org/licenses/by/4.0/">知识共享署名 4.0 国际许可协议</a>
    <span>进行许可。 转载时请注明原文链接。</span>
</div>


  
    <div class="post-nav" style="margin-left:-168px;">
      <div class="post-nav-item post-nav-next">
        
          <span>〈 </span>
          <a href="/2016/10/24/前端优化的技巧/" rel="next" title="前端性能优化问题">
          前端性能优化问题
          </a>
        
      </div>
  
      <div class="post-nav-item post-nav-prev">
          
          <a href="/2017/05/07/Javascript类库作用以及对比/" rel="prev" title="第十七章 JQuery的基础学习">
            第十七章 JQuery的基础学习
          </a>
          <span>〉</span>
        
      </div>
    </div>
  


	
	<div style="width:109%; margin-left:-144px" id="lv-container" data-id="city" data-uid="MTAyMC80OTg5NS8yNjM4Ng==">
	<script type="text/javascript">
   	   (function(d, s) {
       		var j, e = d.getElementsByTagName(s)[0];

       		if (typeof LivereTower === 'function') { return; }

       		j = d.createElement(s);
       		j.src = 'https://cdn-city.livere.com/js/embed.dist.js';
       		j.async = true;

       		e.parentNode.insertBefore(j, e);
   	   })(document, 'script');
	</script>
	<noscript> 为正常使用来必力评论功能请激活JavaScript</noscript>
        </div>
	



    </div>

    

  </div>
  <footer class="footer text-center">
    <div id="bottom-inner">
        <a class="bottom-item" href target="_blank">GitHub</a> |
        <a class="bottom-item" href>友情链接</a> |
        <a class="bottom-item" href="https://hexo.io" target="_blank">Powered by hexo</a> |
        <a class="bottom-item" href="https://github.com/fooying/hexo-theme-xoxo-plus" target="_blank">Theme xoxo-plus</a> |
        <a class="bottom-item" href="/atom.xml">订阅</a>
    </div>
</footer>

  

<script>
  (function(window, document, undefined) {

    var timer = null;

    function returnTop() {
      cancelAnimationFrame(timer);
      timer = requestAnimationFrame(function fn() {
        var oTop = document.body.scrollTop || document.documentElement.scrollTop;
        if (oTop > 0) {
          document.body.scrollTop = document.documentElement.scrollTop = oTop - 50;
          timer = requestAnimationFrame(fn);
        } else {
          cancelAnimationFrame(timer);
        }
      });
    }

    var hearts = [];
    window.requestAnimationFrame = (function() {
      return window.requestAnimationFrame ||
        window.webkitRequestAnimationFrame ||
        window.mozRequestAnimationFrame ||
        window.oRequestAnimationFrame ||
        window.msRequestAnimationFrame ||
        function(callback) {
          setTimeout(callback, 1000 / 60);
        }
    })();
    init();

    function init() {
      css(".heart{z-index:9999;width: 10px;height: 10px;position: fixed;background: #f00;transform: rotate(45deg);-webkit-transform: rotate(45deg);-moz-transform: rotate(45deg);}.heart:after,.heart:before{content: '';width: inherit;height: inherit;background: inherit;border-radius: 50%;-webkit-border-radius: 50%;-moz-border-radius: 50%;position: absolute;}.heart:after{top: -5px;}.heart:before{left: -5px;}");
      attachEvent();
      gameloop();
      addMenuEvent();
    }

    function gameloop() {
      for (var i = 0; i < hearts.length; i++) {
        if (hearts[i].alpha <= 0) {
          document.body.removeChild(hearts[i].el);
          hearts.splice(i, 1);
          continue;
        }
        hearts[i].y--;
        hearts[i].scale += 0.004;
        hearts[i].alpha -= 0.013;
        hearts[i].el.style.cssText = "left:" + hearts[i].x + "px;top:" + hearts[i].y + "px;opacity:" + hearts[i].alpha + ";transform:scale(" + hearts[i].scale + "," + hearts[i].scale + ") rotate(45deg);background:" + hearts[i].color;
      }
      requestAnimationFrame(gameloop);
    }

    /**
     * 给logo设置点击事件
     * 
     * - 回到顶部
     * - 出现爱心
     */
    function attachEvent() {
      var old = typeof window.onclick === "function" && window.onclick;
      var logo = document.getElementById("logo");
      if (logo) {
        logo.onclick = function(event) {
          returnTop();
          old && old();
          createHeart(event);
        }
      }
      
    }

    function createHeart(event) {
      var d = document.createElement("div");
      d.className = "heart";
      hearts.push({
        el: d,
        x: event.clientX - 5,
        y: event.clientY - 5,
        scale: 1,
        alpha: 1,
        color: randomColor()
      });
      document.body.appendChild(d);
    }

    function css(css) {
      var style = document.createElement("style");
      style.type = "text/css";
      try {
        style.appendChild(document.createTextNode(css));
      } catch (ex) {
        style.styleSheet.cssText = css;
      }
      document.getElementsByTagName('head')[0].appendChild(style);
    }

    function randomColor() {
      // return "rgb(" + (~~(Math.random() * 255)) + "," + (~~(Math.random() * 255)) + "," + (~~(Math.random() * 255)) + ")";
      return "#F44336";
    }

    function addMenuEvent() {
      var menu = document.getElementById('menu-main-post');
      if (menu) {
        var toc = document.getElementById('toc');
        if (toc) {
          menu.onclick = function() {
            if (toc) {
              if (toc.style.display == 'block') {
                toc.style.display = 'none';
              } else {
                toc.style.display = 'block';
              }
            }
          };
        } else {
          menu.style.display = 'none';
        }
      }
    }

  })(window, document);
</script>

  



  

</body>
</html>
